Optimal Tag Sets for Automatic Image Annotation

In this paper we introduce the Beam Search CRM (BS-CRM) model. This model implements two novel improvements to the basic CRM [2]. First, we argue that using a Minkowski kernel allows us to capture the covariance of visual features more effectively than the standard Gaussian kernel. Second, we advocate a procedure that selects the most informative subset of tags as the image annotation. Our procedure captures the mutual dependence within a set of tags, and naturally prevents noisy tags from being assigned during the search procedure. In automatic image annotation the basic objective is to find the set of tags w = {w1 . . .wk} that serves as the best annotation for the test image represented with a set of feature vectors f = {~f1. . .~fm}. The traditional approach used by [2] and many subsequent publications [3] [5] [4] involves estimating the marginal probability distribution over individual tags P(w|f) and annotating the image with top-ranked tags from that distribution. This approach however does not take into consideration any correlation between the tags: the top-ranked tags could be incohesive and contradictory, e.g. {tropical, blizzard, supernova}. Beam Search: To address both of the above issues, we propose to annotate images with the most informative subset of tags. We define the amount of information I(w) present in a set of tags w as the expected excess number of bits required to encode this set with the background model: I(w) = P(w|f) · log P(w|f) P0(w) .